Leveraging POMDPs Trained with User Simulations and Rule-based Dialogue Management in a Spoken Dialogue System

We have developed a complete spoken dialogue framework that includes rule-based and trainable dialogue managers, speech recognition, spoken language understanding and generation modules, and a comprehensive web visualization interface. We present a spoken dialogue system based on Reinforcement Learning that goes beyond standard rule based models and computes on-line decisions of the best dialogue moves. Bridging the gap between handcrafted (e.g. rule-based) and adaptive (e.g. based on Partially Observable Markov Decision Processes POMDP) dialogue models, this prototype is able to learn high rewarding policies in a number of dialogue situations. 1 Reinforcement Learning in Dialogue Machine Learning techniques, and particularly Reinforcement Learning (RL), have recently received great interest in research on dialogue management (DM) (Levin et al., 2000; Williams and Young, 2006). A major motivation for this choice is to improve robustness in the face of uncertainty due for example to speech recognition errors. A second important motivation is to improve adaptivity w.r.t. different user behaviour and application/recognition environments. The RL approach is attractive because it offers a statistical model representing the dynamics of the interaction between system and user. This contrasts with the supervised learning approach where system behaviour is learnt based on a fixed corpus. However, exploration of the range of dialogue management strategies requires a simulation environment that includes a simulated user (Schatzmann et al., 2006) in order to avoid the prohibitive cost of using human subjects. We demonstrate various parameters that influence the learnt dialogue management policy by using pre-trained policies (section 5). The application domain is a tourist information system for accommodation and events in the local area. The domain of the trained DMs is identical to that of a rule-based DM that was used by human users (section 4), allowing us to compare the two directly. 2 POMDP demonstration system The POMDP DM implemented in this work is shown in figure 1: at each turn at time t, the incomingN user act hypotheses an,u split the state space St to represent the complete set of interpretations from the start state (N=2). A belief update is performed resulting in a probability assigned to each state. The resulting ranked state space is used as a basis for action selection. In our current implementation, belief update is based on probabilistic user responses that include SLU confidences. Action selection to determine system action am,s is based on the best state (m is a counter for actions in action set A). In each turn, the system uses an -greedy action selection strategy to decide probabilistically if to exploit the policy or explore any other action at random. (An alternative would be softmax, for example.) At the end of each dialogue/session a reward is assigned and policy entries are added or updated for each state-action pair involved. These pairs are stored in tabular form. We perform Monte Carlo updating similar to (Levin et al., 2000): Qt(s, a) = R(s, a)/n+Qt−1 · (n− 1)/n (1) where n is the number of sessions, R the reward and Q the estimate of the state-action value. At the beginning of each dialogue, a user goal UG (a set of concept-value pairs) is generated randomly and passed to a user simulator. The user simulator takes UG and the current dialogue context to produce plausible SLU hypotheses. These Turn
t1
 POLICY:

 Q(s1,a1)
 s1,a1
 s1,a2
 s2,a1